go/oasis-node/cmd/storage: Add command that flattens consensus dbs#6311
Merged
martintomazic merged 4 commits intomasterfrom Oct 20, 2025
Merged
go/oasis-node/cmd/storage: Add command that flattens consensus dbs#6311martintomazic merged 4 commits intomasterfrom
martintomazic merged 4 commits intomasterfrom
Conversation
✅ Deploy Preview for oasisprotocol-oasis-core canceled.
|
martintomazic
commented
Sep 3, 2025
martintomazic
force-pushed
the
martin/feature/cli/compact-db-instances
branch
2 times, most recently
from
5fd1136 to
4d3e8f9
Compare
kostko
reviewed
Sep 3, 2025
martintomazic
commented
Sep 4, 2025
martintomazic
force-pushed
the
martin/feature/cli/compact-db-instances
branch
2 times, most recently
from
9a12d7f to
c07cc2b
Compare
martintomazic
commented
Oct 2, 2025
Comment on lines
+299
to
+301
| if err := cmdCommon.Init(); err != nil { | ||
| cmdCommon.EarlyLogAndExit(err) | ||
| } |
Contributor
Author
There was a problem hiding this comment.
Commands above don't initialize this, thus all the logs are lost as logger is not initialized. Note this redirects logs to stdout, which depending on the configuration may produce a lot of logs on the operator terminal.
Should this be actually written to stderr by default?
update: related to #6311 (comment)
martintomazic
commented
Oct 2, 2025
martintomazic
commented
Oct 2, 2025
martintomazic
changed the title
kostko
reviewed
Oct 7, 2025
martintomazic
force-pushed
the
martin/feature/cli/compact-db-instances
branch
from
7ab7813 to
66fe975
Compare
Contributor
Author
|
Added a docs section as requested yesterday. |
martintomazic
force-pushed
the
martin/feature/cli/compact-db-instances
branch
from
66fe975 to
c4616bb
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6311 +/- ##
==========================================
- Coverage 64.82% 64.19% -0.63%
==========================================
Files 698 698
Lines 67826 67908 +82
==========================================
- Hits 43968 43594 -374
- Misses 18860 19297 +437
- Partials 4998 5017 +19 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
martintomazic
force-pushed
the
martin/feature/cli/compact-db-instances
branch
from
c4616bb to
9c2b25b
Compare
kostko
approved these changes
Oct 13, 2025
martintomazic
force-pushed
the
martin/feature/cli/compact-db-instances
branch
from
9c2b25b to
ac4c227
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #6309.
This command has been thoroughly tested and is able to immediately reclaim all the disk space even if enabling pruning late.
Unfortunately, there is a known corner case where disk space used during the peak of the compaction may be double the original size - see. It is for this reason that I suggest to release this as experimental command.
Performance:
Currently it takes less then
1hto compact all the consensus databases (with all the state/blocks from the genesis) on my2TBplayground.I haven't observed that increasing number of the
Flattenworkers would increase the overall disk usage during the peak of the compaction (as warned by the badger cli documentation). How about adding this as an optional parameter?This way users with spare disk space could e.g. configure it to use 8 workers, and thus speed-up compaction significantly.
3-5times as from my experiments.We may also explore the impact of increasing
BlockCacheSize...Follow-up
Given that I was testing this only on the consensus DB instances, I suggest to add compaction of the runtime DB instances as a follow-up. The testing there is also complicated as runtime state pruning is very slow, making it hard to prepare realistic samples.
I suggest the following order of follow-ups:
compactcommand.